Problem statement

ListingTable currently loads all statistics for Parquet files when an external table is created. This makes sense in the context of how ClickBench counts table loading (statistics loading at table initialization is not counted towards query time) but it has drastic problems for other systems:

• The TableProvider has no idea what statistics are useful or not. All it knows is what columns are being projected and filtered on. But e.g. for select * from t you want to get no stats but for select * from t order by col you'd want to get just stats for col.
• Real world systems can have multi TB tables with tens of thousands of files and serve multi-tenant queries so they have thousands of tables. Collecting stats upfront is not possible. It also results in memory bloat: holding stats for all columns on a wide table for 100k files would itself be larger than memory. Even with a limited size cache (see e.g. Add limit to DefaultFileStatisticsCache #19052) you'd end up just churning the cache.
• For systems with an external metadata store / catalog that can provide more granular statistics requesting unnecessary stats is wasted work. For Parquet it is more or less a sunk cost (you have to do the IO for the entire footer) but if your system is able to do more granular IO it becomes wasted work.

Proposed solution

Long term I'd like to move towards #21996.
This is the first minimal step to enable it: allow statistics requests to be bound to a scan so that a TableProvider can know what statistics to gather.

It still needs to populate the dense / wasteful statistics data structures, ListingTable doesn't use this and there is no optimizer rule to decide what stats should be collected. Users have to write this all themselves, but at least now it will be possible. Currently it is not possible because there is no way to communicate from a logical optimizer rule into a TableScan::scan_with_args call.

Alternatives considered

Make this an opaque extension and put extensions: Extensions onto ScanArgs.
I think the API being proposed here is reasonable and has the opportunity to grow into something that DataFusion actually uses. Erasing the types / entire API obfuscates the functionality, I'd rather not do that.

Next steps

Two straightforward followups would be to:

1. Add a logical optimizer rule to populate statistics requests by introspecting plans. I think there are a lot of obvious cases that could be covered, and users can always add their own.
2. Have ListingTable use it. I spoke with @alamb about this and it's not clear that this will be a win: DataFusion implicitly creates a table for a query like select col from 't.parquet'; and at table creation time we do not know the query. I'd argue that if we are collecting stats for a single query we should tailor them to that query, but I also concede that there isn't that much benefit in terms of runtime given the sunk cost for Parquet (there is still a memory usage benefit). Since we don't use ListingTable I am unlikely to push this.

Which issue does this PR close?

• Related to Query-aware statistics requests via ScanArgs / ScanResult (RFC for #21624) #21996
• Related to datafusion.execution.collect_statistics on wide tables #21624

What changes are included in this PR?

Five small, independently-reviewable commits:

1. refactor: add TableScanBuilder, deprecate TableScan::try_new — TableScan::try_new takes five positional args and bare TableScan { .. } literals are fragile to field additions. Introduce TableScanBuilder (with From<TableScan>), move schema derivation into build(), deprecate try_new (delegates to the builder), migrate all in-tree callers. Pure refactor.
2. feat: add StatisticsRequest — new public vocabulary types in datafusion-expr-common::statistics. Nothing consumes them yet. I left out any sort of response / result types.
3. feat: add TableScan::statistics_requests field — an advisory Vec<StatisticsRequest> on TableScan, settable via TableScan::with_statistics_requests / TableScanBuilder. Empty by default; DataFusion's own rules never populate it.
4. feat: thread statistics requests into ScanArgs — ScanArgs gains statistics_requests; the physical planner threads TableScan::statistics_requests into it so the request reaches TableProvider::scan_with_args.
5. test: e2e statistics-request flow via a custom optimizer rule — an integration test playing both external roles.

Deliberately left out vs #21996: the built-in RequestStatistics optimizer rule, the FilePruner / ListingTable consumer integration, the PartitionedFile::satisfied_stats per-file response field, and StatisticsValue::Distribution (which would depend on the now-deprecated Distribution type).

Are these changes tested?

Yes:

• datafusion-expr-common: a unit test that StatisticsRequest is hashable / usable as a HashMap key.
• datafusion/core/tests/user_defined/statistics_requests.rs: an end-to-end integration test where a custom OptimizerRule annotates TableScan and a custom TableProvider asserts the requests reach scan_with_args — plus a test that without such a rule the provider sees an empty request list.
• All existing datafusion-expr / datafusion-optimizer / datafusion-proto tests pass against the TableScanBuilder refactor.

Are there any user-facing changes?

Yes — this needs the api change label:

• New public types StatisticsRequest, StatisticsValue, SatisfiedStatistics (re-exported via datafusion_expr::statistics).
• New TableScanBuilder; TableScan::try_new is deprecated (still works, delegates to the builder).
• TableScan gains a new public field statistics_requests — this breaks exhaustive TableScan { .. } struct literals downstream (the recommended fix is TableScanBuilder).
• ScanArgs gains with_statistics_requests / statistics_requests.

🤖 Generated with Claude Code